12. Demo: Training a DQN Model in Python

Part 1

Cd13650 C5 L3 Demo 4 Part 1 V2

Training Algorithm Implementation: Part 1

  • Dataset Preparation:

    • Train/test split at 80/20 ratio.
    • Dataset of 138 training examples with 5 features.
    • Understanding feature-column mapping for accurate processing.

  • Initializations:

    • Define column indexes for feature tracking.
    • Track the length of the training set and define batch size and episodes.

  • Training Episode Structure:

    • Start by initializing key variables: state, total profit, and trade counts.
    • Use helper functions to obtain state and iterate over the dataset length.
    • Execute trading actions (buy/sell) based on state analysis.

  • Additional Variables:

    • Inventory management for trading actions.
    • Real-time tracking of episode progress and profits using progress indicators and reward calculations.

Part 2

Cd13650 C5 L3 Demo 4 Part 1b V2

Training Algorithm Implementation: Part 2

Buy Actions:

  • Action 0: Hold – Keep current position unchanged.
  • Action 1: Buy
    • Buy Price: Based on the closing price at the current timestep (t).
    • Inventory Update: Record the buy price in the agent's inventory.
    • Tracking: Append the timestep to the states_buy list for marking buy actions.
    • Monitoring: Print a message indicating a purchase has occurred.

Sell Actions:

  • Only sell if inventory is non-empty.
  • Action 2: Sell
    • Retrieve Buy Price: Use pop to extract the earliest buy price from inventory.
    • Sell Price: Acquire the current closing price from the dataset.
    • Profit Calculation: Subtract the buy price from the sell price.
    • Reward Rule: Positive profit results in a reward, zero otherwise.
    • Profit Tracking: Record total profit by adding the trade profit to cumulative gains.
    • Trade Monitoring: Update total winners and losers count, and append to states_sell for visualization.
    • Live Tracking: Display the selling price and profit for each trade.

Part 3

Cd13650 C5 L3 Demo 4 Part 1c V2

Training Algorithm Implementation: Part 3

  1. Define Actions & Set Training Flags:

    • Establish a condition to mark training completion when reaching dataset length.
    • Proceed without checking this until necessary.

  2. Update Agent's Memory:

    • Append data such as state, action, reward, next state, and training completion status.

  3. Statistics & Reporting:

    • Display metrics like total profit, episode number, winners and losers count, max loss, and total loss to evaluate neural network performance.

  4. Plotting Functions:

    • Utilize predefined functions to visualize trades using data like close price, Bollinger Bands, time states for buy/sell actions, and profits.

  5. Experience Replay & Loss Tracking:

    • Ensure agent memory exceeds batch size to execute experience replay and record training losses.
    • Append and sum losses to track changes over batches, plotting loss trends for relevant loops.

  6. Model Saving:

    • Save the model with episode-specific identifiers to ensure progress documentation.

Part 4

Cd13650 C5 L3 Demo 4 Part 2 V2

Training Algorithm Implementation: Part 4

  • Episode Output Anallysis:

    • Completed Episode 0 with a total profit of $9.
    • Recorded $12 in winning trades and $3 in losing trades.
    • Maximum individual loss was approximately $2.5, while total loss accumulated to $93.

  • Visual Representation:

    • A trades chart displays key data:
      • Black Line: Price action
      • Red Triangles: Buy signals
      • Green Triangles: Sell signals
      • Blue Bands: Boundary markers

  • Performance Insights:

    • Initial training loss showed a decreasing trend, indicating some learning.
    • Comparison to previous episode shows reduction in total and maximum loss, yet fewer trades were made, affecting profitability.

  • Observations:

    • Model has potential but needs more extensive training.
    • Initial findings should be validated by further tests and longer training periods to enhance efficiency.

  • Next Steps:

    • Further episodes will help in observing learning patterns.
    • Validation test suggested to ensure the model isn't overfitting.

Part 5

Cd13650 C5 L3 Demo 4 Part 3 V2

Testing the Trained Model

Understanding the steps to prepare a model for testing and validation is crucial for effective machine learning. Here’s a concise guide:

  • Parameter Reconfiguration: Adjust the dataset length for testing and reset flags to ensure a fresh start.

  • Initialization:

    • Set up new lists for time tracking.
    • Re-instantiate the pre-trained model for evaluation.
    • Load the model using specific window size and features.

  • Transition from Training to Testing:

    • Adapt training sequences to testing, swapping training variables with their test counterparts.
    • Eliminate elements linked to model fitting and training loss.

  • Testing Loop Preparation:

    • Initialize states and clear previous inventories.
    • Iterate through the test data without fitting to ensure genuine evaluation.

  • Adjustment and Check:

    • Ensure all variables and flags align with the test environment.
    • Employ the extended test set for more data if necessary to trigger model actions.

  • Progressing to Backtesting: After validating, learn to implement a backtesting process to optimize the model further in subsequent lessons.

Next Concept